Dynamic scaling for a robotic surgical system

ABSTRACT

A robotic surgical system in which the system applies a scaling factor between user input from a user input device and corresponding movements of the robotic manipulator. Scaling factors may be applied or adjusted based on detected conditions such as the type of instrument being manipulated, detected distance between multiple instruments being manipulated, user biometric parameters.

This application is a continuation of U.S. application Ser. No.16/931,425, filed Jul. 16, 2020, which claims the benefit of each of thefollowing US Provisional Applications: U.S. 62/874,960, filed Jul. 16,2019, U.S. 62/874,967, filed Jul. 16, 2019, 62/874,969, filed Jul. 16,2019 and 62/874,974, filed Jul. 16, 2019.

BACKGROUND

Surgical robotic systems are typically comprised of one or more roboticmanipulators and a user interface. The robotic manipulators carrysurgical instruments or devices used for the surgical procedure. Atypical user interface includes input devices, or handles, manuallymoveable by the surgeon to control movement of the surgical instrumentscarried by the robotic manipulators. The surgeon uses the interface toprovide inputs into the system and the system processes that informationto develop output commands for the robotic manipulator. Some systemsallow the user to optimize the ergonomics of the interface by“clutching,” which means temporarily disabling output motion at thesurgical instrument in response to movement of the input device, toallow the surgeon to move the input device to a position that allows thesurgeon to more comfortably manipulate the handle.

Some systems may include predetermined or user-selectable motionscaling, in which a scaling factor is applied between the velocity ofmotion of the user input given at the input devices and the resultingvelocity at which the corresponding robotic manipulator moves thesurgical instrument. Surgeons may desire a fine scaling motion forcertain procedures or steps, while in others s/he may prefer largermotion, relative to the movement of the user interface.

Motion scaling enables a surgeon to increase precision (fine roboticmotions for large user input device motions) or increase range of motionand surgeon comfort (large robotic motions for small user input devicemotions). Motion scaling and the goal of surgeon comfort also give riseto the concept of clutching, wherein a surgeon repositions thegrips/handles of the input devices while robotic motion is disengaged.Reducing the need for clutching may be desirable for some users sinceclutching can add time to the procedure.

Highly scaled-down motion (i.e. fine robotic motion for large inputdevice motions) requires frequent clutching and increases the surgeon'seffort, due to the fact that the surgeon moves the user input devicesessentially twice as often as the robot moves the instrument.

In some surgical systems, the user has the ability to select differentlevels of motion scaling to adjust the mapping of the magnitude ofmotion at the input device to the magnitude of motion at the outputdevice. This functionality can be useful to enable the user to adjustthe sensitivity of the system to motion to better suit differentsurgical situations depending on the level of precision required. Withthese systems, users select the motion scaling setting from a menu,which requires them to stop performing surgical tasks to make a change.This results in a delay in the procedure as the user switches scalingsettings between a high precision task in a small workspace in the bodycavity and another task requiring larger motions. The concepts describedin this application improve the efficiency of the surgical procedure byenabling dynamic scaling in which the motion scaling settings areadjusted during use either autonomously or by the user without requiringthe procedure to pause. Ideally, this solution will allow themanipulator to move as the user wants and expects over a wide variety oftasks without any need for the user to think about or potentially evennotice changes in the motion scaling.

This application describes a system and method configured to tune themotion scaling applied between the input given at the user input deviceand the resulting motion of the surgical instrument by the roboticmanipulator. In some embodiments, the system automatically tunes themotion scaling as appropriate for the current surgical context. In someembodiments, the system predicts the desired motion scaling factor basedon the instrument that is or has been mounted to a robotic manipulator,and/or based on the task or tasks the system determines are beingcarried out or are expected to be carried out by the user. In otherembodiments, the system automatically varies a scaling factor during thecourse of a surgical procedure based on input to the system regardingthe procedure or steps being performed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of a robot-assisted surgical system;

FIG. 2 shows a functional block diagram illustrating a first method ofdetermining a scaling factor based on instrument type;

FIG. 3 shows a functional block diagram illustrating a second method ofdetermining a scaling factor based on instrument type;

FIG. 4 shows a functional block diagram illustrating a third method ofdetermining a scaling factor based on instrument type;

FIG. 5 is a process flow diagram illustrating flow for the thirdexemplary embodiment.

FIGS. 6A-6C show functional block diagrams illustrating methods ofdetermining a scaling factor based on instrument distance.

FIG. 7 shows a display displaying an intraoperative image of thesurgical work site captured by the camera, and an eye tracking unitbeneath the display, and further illustrates use of the system accordingto the FIGS. 6A and 6B embodiments;

FIG. 8 shows a display displaying an intraoperative image of thesurgical work site captured by the camera, and further illustrates useof the system according to the FIG. 6C embodiment;

FIG. 9 schematically illustrates process flow for the FIG. 6A-6C.

DETAILED DESCRIPTION

Although the inventions described herein may be used on a variety ofrobotic surgical systems, the embodiments will be described withreference to a system of the type shown in FIG. 1. In the illustratedsystem, a surgeon console 12 has two input devices such as handles 17,18. The input devices 12 are configured to be manipulated by a user togenerate signals that are used to command motion of a roboticallycontrolled device in multiple degrees of freedom. In use, the userselectively assigns the two handles 17, 18 to two of the roboticmanipulators 13, 14, 15, allowing surgeon control of two of the surgicalinstruments 10 a, 10 b, and 10 c disposed at the working site at anygiven time. To control a third one of the instruments disposed at theworking site, one of the two handles 17, 18 is operatively disengagedfrom one of the initial two instruments and then operatively paired withthe third instrument. A fourth robotic manipulator, not shown in FIG. 1,may be optionally provided to support and maneuver an additionalinstrument.

One of the instruments 10 a, 10 b, 10 c is a camera that captures imagesof the operative field in the body cavity. The camera may be moved byits corresponding robotic manipulator using input from a variety oftypes of input devices, including, without limitation, one of thehandles 17, 18, additional controls on the console, a foot pedal, an eyetracker 21, voice controller, etc. The console may also include adisplay or monitor 23 configured to display the images captured by thecamera, and for optionally displaying system information, patientinformation, etc.

The input devices 17, 18 are configured to be manipulated by a user togenerate signals that are processed by the system to generateinstructions used to command motion of the manipulators in order to movethe instruments in multiple degrees of freedom. A control unit 30 isoperationally connected to the robotic arms and to the user interface.The control unit receives the user input from the input devicescorresponding to the desired movement of the surgical instruments, andgenerates commands to cause the robotic manipulators to maneuver thesurgical instruments accordingly.

Where the system includes an eye tracker, input from the eye tracker 21may be used to determine the direction of the user's gaze towards thedisplayed image, and signals are generated to move the manipulatorholding the camera to move the camera towards the site within thepatient corresponding to the region of the user's focus. Input from theeye tracker may also be used to determine an increase or decrease in thedistance between the user's eyes and the display, and signals aregenerated to move the manipulator holding the camera away from or closerto, respectively, the site being observed by the camera. See commonlyowned U.S. Pat. No. 9,360,934, which is incorporated herein byreference. Naturally, similar eye tracking input may be used to pan orzoom the image without moving the manipulator supporting the camera,such as by articulating the camera itself, performing digital pan/zoometc.

One or more of the degrees of freedom of the input devices are coupledwith an electromechanical system capable of providing gravitycompensation for the user input, and/or providing haptic feedback to thesurgeon. It should be understood that the concepts described in thisapplication are not limited to any particular user input deviceconfiguration. Alternative configurations include, without limitation,those described in co-pending application Ser. No. 16/513,670 entitledHAPTIC USER INTERFACE FOR ROBOTICALLY CONTROLLED SURGICAL INSTRUMENTS(Atty Ref: TRX-10610, incorporated herein by reference), and userinterfaces or haptic devices known to those of skill in the art ordeveloped in the future.

The surgical system allows the operating room staff to remove andreplace surgical instruments carried by the robotic manipulator, basedon the surgical need. Once instruments have been installed on themanipulators, the surgeon moves the input devices to provide inputs intothe system, and the system processes that information to develop outputcommands for the robotic manipulator in order to move the instrumentsand, as appropriate, operate the instrument end effectors. The userinterface may be one that allows the surgeon to select motion scalingfactors as well as to clutch and reposition the handles to a morecomfortable position. In some cases, the surgeon may desire a finescaling motion, while in others s/he may prefer larger motion, relativeto the movement of the user interface. These desires may besituation-dependent and may be predictable based on the procedure andthe sequence of installed instruments.

Instrument Type Based or Context-Based Motion Scaling

In a first embodiment, the system is configured to identify the type ofsurgical instrument that is attached to, or approaching for attachmentto, one of the system's manipulators. For this purpose, the systempreferably includes a sensor 32. The sensor is positioned to determinethe type of instrument that a user is in the process of engaging with amanipulator, or that the user has engaged with the manipulator. Thelocation of the sensor depends on the type of sensor used. It may be onthe manipulator, in proximity to the manipulator within range to senseor read or information carried by or on the surgical instrument or tosense, detect, or read a property of the surgical instrument. Withoutlimiting the scope of the claims, the following are non-limitingexamples of sensors that may be used for this purpose. Each of these maybe positioned on the relevant robotic manipulator, or in proximity tothe manipulator:

(i) an RFID reader/antenna or near field communication sensor, thatreads information relating to the type, serial number, geometry, orother identity-related information, that is electronically stored on aportion of the instrument or a component attached to the instrument. Inthe RFID context the device storing the information is sometimesreferred to as an “RFID tag.”

(ii) a circuit that reads such information from a memory device that iscarried by the instrument when the instrument is placed in electroniccommunication with the circuit upon its mounting to the manipulator;

(iii) an optical sensor, optical scanner, or image sensor. Such a sensormay generate an output based on a property of the instrument or a deviceattached to the instrument (e.g. a code, symbol, marker, text, color ofor on the surgical instrument) using, in some cases, computer vision orimage processing techniques. In some embodiments, a laparoscopic camerapositioned within the patient's body cavity, and optionally supported bya robotic manipulator, may be used to capture images of the portion ofthe instrument disposed within the body, and computer vision or imageprocessing techniques then used to determine a code, symbol, marker,text, colors, shapes etc. of or on the surgical instrument;

(iv) a sensor that detects when mechanical components on the manipulatorarm are moved by the mounting of instruments, for example in aconfiguration where each different type of instrument produces adifferent amount or type of movement of the mechanical components whenmounted. See for example concepts for giving mechanical input of thetype described in U.S. application Ser. No. 15/975,965, Instrument EndEffector Identification, which is incorporated herein by reference.

The system is optionally configured to further receive input orinformation that identifies, or from which the processor can identify,the surgical procedure that is to be performed. Any type ofcommunication device used to electronically give input to the system canbe used to identify the type of procedure to the system. This mightinclude a graphical user interface together with an input device such asa touch screen, head tracker, mouse, eye tracker 21 (e.g. allowing theuser to look at an icon, image, words, etc. and confirm a selectionusing another input device such as a foot pedal or switch at the surgeonconsole).

The system includes at least one processor (which may be part of controlunit 30) and at least one memory. The memory stores instructionsexecutable by the one processor to perform the various functionsdescribed herein. The memory also includes a database storing scalingfactors. Within the database, are predetermined scaling factors forvarious instrument types. For example, a particular scaling factor mightbe correlated with a first instrument type (e.g. a Maryland dissector)and a second, different, scaling factor might be correlated with asecond instrument type (e.g. a needle holder). If it is beneficial for agiven instrument type to operate using a different scaling factor fordifferent procedures, the scaling factor data may be further correlatedby procedure type. Scaling factors might be further correlated based onany combination of the following factors: procedure type (i.e. the typeof surgery being performed), task type (i.e. the type of surgical stepsthat instrument can be used for), particular users (in other words, eachuser's preferred scaling factor for that instrument type, and/or forthat instrument type for a particular procedure, and/or for thatinstrument type for a particular surgical task), etc. The database mightfurther include data correlating surgical procedures to typicalsequences of instrument exchanges and/or tasks that typically occurduring that procedure. This may be further correlated to both proceduretype and user, since different users may use different sequences ofinstrument exchanges for the same procedure.

The database further includes data correlating input of the type thatmay be received from the sensor 32 with the various types of instrumentsavailable to be mounted to the robotic manipulator, allowing the systemto determine based on the input what type of instrument has been, or isin the process of being, mounted to the manipulator.

In one embodiment depicted in FIG. 2, the appropriate scaling factor isdetermined based solely on the instrument type as determined based onthe sensor input. The system receives input from the sensor 32 as theinstrument is moved in proximity to, or is mounted to, the correspondingrobotic manipulator. Based on the input, the system determines the typeof instrument, and determines from the database the type of scalingfactor to be used for that type of instrument. The determined scalingfactor is applied between the user input given at the relevant inputdevices (e.g. input device 17 or 18) and the resulting movement of theinstrument by the robotic manipulator.

In a slightly modified version of the FIG. 2 embodiment, the systemfurther receives user identification input. This may be done using anyconventional form of input, such as touch screen, keyboard, mouse, scanof a user's identification wristband/badge, etc., any of the forms ofinput devices mentioned above. In some embodiments, the console mayinclude sensor that allow the system to recognize the user based onbiometric parameters sensed using sensors at the console (e.g. imagesensors for facial recognition, eye scanners, fingerprint scanners,voice sensors). In this embodiment, scaling factors associated with thatuser and the relevant instrument type are retrieved from the databaseand used to control instrument motion.

The embodiment depicted in FIG. 3 is similar to that shown in FIG. 2,but additionally includes a step in which the system receivesprocedure-type input, which may be input to the system using anysuitable type of input device. Use of the robotic system incorporatingthis feature might include an initialization step in which the useridentifies the intended surgical procedure to the system or selects thesurgical procedure from options displayed at the console. In otherembodiments, the system may determine the procedure type. For example,the database may include data correlating procedure types with certainaspects of the surgical set-up, such as which types of instruments areon each of the arms, the arrangement of the robotic manipulatorsrelative to the patient bed (e.g. as determined by applying computervision to images of the robotic manipulators and the patient bed, or thearrangement of instruments or trocars within the patient (e.g. asdetermined by applying computer vision to images of the patient once thetrocars have been positioned or the instruments inserted through thetrocars). Scaling factors associated with that procedure type and therelevant instrument type are thus retrieved from the database. As withthe FIG. 2 embodiment, the system may further receive useridentification input, and scaling factors associated with that user, theidentified procedure type and the relevant instrument type are retrievedfrom the database and used to control instrument motion.

FIG. 4 illustrates an embodiment similar to FIG. 3, but in which, basedon the identified surgical procedure and user, the processor determinesusing the database the typical sequence of instrument exchanges for thatprocedure or procedure and user, excluding any deviations or exceptions.For example, a cholecystectomy would typically begin with a grasper forretracting the fundus of the gallbladder, a second grasper and aMaryland grasper for assisting in the dissection of the triangle ofcalot. During the course of the procedure, as the processor receivesinput from the sensor in response to movement of an instrument intoproximity or engagement to the manipulator, the processor obtains thescaling factor for that instrument (and, if applicable, that instrumentand procedure) then adjusts the motion scaling of the system to anoptimal ratio for the task at hand. This ratio could also factor instored surgeon/user preferences, which would also be stored in thedatabase.

The process is repeated each time an instrument exchange occurs. Thus,in the current example, after the Maryland grasper has been removed anda clip applier inserted, the system may adapt the motion scaling again,to an optimal ratio for the next task it anticipates that the user willperform. As the procedure continues and instruments are exchanged forone another, the system constantly updates the motion scaling factoraccording to what perceived step the surgeon is performing in the namedprocedure. If an instrument exchange is performed that produces anatypical combination or sequence of instruments for that procedure (i.e.the detected instrument is one not expected based on the procedure), thesystem may default to a standard motion scaling factor. The system maybe further equipped to allow the user to override the dynamic scalingfunction or the scaling factor applied for a given step using inputdevices (e.g. the GUI, a dial on the input device 17, 18) at the surgeonconsole.

In a slightly modified embodiment, the concepts described above might beenhanced by task recognition performed using input from the camera 10 bpositioned in the body cavity. Thus, for example, if the systemrecognizes one, or a sequence of, tasks being performed that differsfrom the expected sequence, the processor might determine the procedurebeing performed based on the task or sequence of tasks, using datastored in the database correlating, for example, patterns or sequencesof instrument movements or actuations with tasks. The processor may thencause the user to be advised of the determined procedure and prompted toconfirm using an input device and/or the GUI that the determinedprocedure is being performed. If confirmed, the processor applies thescaling factors for the newly determined procedure, based on theinformation stored in the database.

While this discussion focuses on motion scaling, motion scaling is butone system feature that could be adjusted based on procedure type andinstrument recognition. In another embodiment, the processor could beconfigured to adjust the functions controlled by various input devices(e.g. foot pedals, buttons, knobs or switches on the surgeon console orinput handles) based on the installed instrument. For example, if amonopolar instrument is installed, a pair of foot pedals may be enabledwith coagulation and cut functions attached, respectively. However, ifthe monopolar instrument is replaced with a suction irrigation device,the foot pedals may switch functionalities to flush and vacuum,respectively. As another example, upon determining that a stapler isbeing or has been mounted to a manipulator, the foot pedal may enabledriving of the sled in the stapler end effector. The basis for thisembodiment is to use the procedure type and installed instrument toautomatically adjust the system for optimal performance, including, butnot limited to, the aforementioned features of motion scaling andbutton/input device task assignment.

The system and method of the FIG. 4 embodiment offer the advantages ofanticipating the next operative step in a procedure, based on theexchange of instruments and concurrent assignment to each arm, and theautomatic adjustment of system features including motion scaling andbutton/input device function assignments according to the instrumentsattached to each robotic manipulator, or as they are assigned to eachhandle on the user interface. The FIG. 5 flow diagram illustrates theprocess flow for this type of system adjustment, based on procedure typeand the instruments installed on the robotic manipulator. Note that asystem may be configured to perform just one of the described functions(adjustment of scaling factors; adjustment of input device functionalityetc.), or multiple ones based on the procedure type, instrument type,and (optionally) user identity or preference.

Scaling Based on Surgical Instrument Distance Data

In other embodiments, scaling factors may be dynamically adjusted basedon information determined by the system relating to the operative sitein the patient anatomy.

In these embodiments, the system includes features that allow the systemto determine the distance between instrument end effectors disposed inthe operative field, and that then adjust the amount of motion scalingbased on that distance.

In these embodiments, the system includes a source of input to thesystem from which a determination of the distance between the endeffectors of the surgical instruments can be made. The source mayinclude input from sensors monitoring joint positions of the roboticmanipulator so as to allow the position of a portion of the end effector(e.g. its tip) to be determined using kinematics. Alternatively, or inaddition, the source may include input from a visual recognition systemthat recognizes the end effectors in the captured image data, allowingthe distance between the end effectors to be calculated. For example,the camera 10 b carried by the robotic manipulator may capture images ofthe distal ends of instruments 10 a and 10 c. As yet another examplesuitable for certain surgical systems, the input may instead oradditionally come from sensors measuring joint positions at the userinterface, and it is then used to determine the end effector positionbased on the end effector position that has been commanded using theuser input device.

The system includes a processor coupled to a memory. The memory storesinstructions executable by the processor to receive the input, determinethe distance between the end effectors, apply a scaling factor based onthe determined distance, and command motion of the robotic manipulatorbased on input from the user input device, as scaled using the appliedscaling factor.

For example, referring to FIG. 6A and FIG. 7, in one embodiment thesystem uses the image data or kinematic data to determine the distancebetween two or more instruments 10 a, 10 b (Vector A). A memoryassociated with the processor includes a database that stores scalingfactors based on instrument distances. After determining the distance,the system obtains the relevant scaling factor, and adjusts or maintainsthe motion scaling based on this distance. Note that this scaling factormay be applied to either or both of the surgical instruments. In oneapplication, when the surgeon is working in tighter spaces and theinstrument tips are determined to be at or below a predeterminedseparation distance D, the system would reduce the motion scaling (e.g.movement of the relevant user interface 17 or 18 in a degree of freedomby a distance X would produce a corresponding motion at the endeffector, in that degree of freedom, that is smaller than X), helpingthe surgeon to work with enhanced precision. Conversely, when thesurgeon is working with instruments determined to be separated by adistance greater than D, the motion scaling would enable larger scalemotions.

While in the above example, the system operates using two scalingfactors (one of which may be 1), the instructions stored in memory foreach of the described embodiments may include a plurality of scalingfactors, each to be applied when the separation between the endeffectors is within a different, predetermined, range. For example, theinstructions might cause a first scaling factor to be applied when theend effectors are separated by a first distance range, a second scalingfactor applied when the end effectors are separated by a second distancerange, and a third scaling factor applied when the end effectors areseparated by a third distance range.

In other embodiments, the processor calculates the scaling factor as afunction of the determined distance, such that the scaling factordynamically changes as the instruments are moved towards or away fromone another.

The instructions optionally would be programmed to cause the processorto use a filter on the distance between instrument tips (i.e. movingaverage) to ensure that changes in motion scaling are smooth and fluid,and to avoid multiple scaling factor changes in short succession.Feedback to the user may be optionally given to notify the user of achange in the scaling factor. For example, a textual or graphicalindication may be displayed on the image display, an auditory alert maybe sounded, or a tactical alert may be generated, such as a vibration ofthe user input handle or another feature that is in contact with theuser (e.g. the seat at the surgeon console). The alert might optionallyprompt the user to give input to the system affirming that the surgeonwould like the change in scaling factor to be applied. The affirmatoryinput may be given from any type of device that could deliver an inputto the system. Non-limiting examples include an input control on thehandle or elsewhere on the console or robotic system (as non-limitingexamples, a switch, button, knob, key, lever, or touch input), a voiceinput device configured to receive verbal input, an eye tracker, headtracker, foot pedal etc.

The embodiment depicted in FIG. 6B and FIG. 7 performs a dynamic scalingfunction similar to the first embodiment, but changes in scaling arebased on the distance between the location the surgeon is viewing on theimage display and the instrument location as displayed on the displayimage, with the understanding that a close distance would indicate workthat would benefit from fine scaling and a larger distance would notnecessitate as a high a level of precision. In this embodiment therobot-assisted surgical system is equipped with a system capable ofmeasuring the direction of the gaze of the surgeon towards the displayat the surgeon console (e.g. eye tracker 21 in FIGS. 1 and 2) as well asa visual recognition system configured to identify the instrument endeffectors in the intraoperative view. The processor operates accordingto a set of instructions in the memory to receive input from the visualrecognition system and from the eye tracker, and to monitor the distancebetween the surgeon's gaze and the displayed instrument tip or endeffector. For scenarios with more than one instrument, the system mayeither use the closest instrument tip to the surgeon's gaze or determinethe centroid of the instruments within the view and monitor the distancebetween the centroid and the surgeon's gaze (Vector B in FIG. 6C). Aswith the FIG. 6A embodiment, based on this dimension monitored by theprocessor, dynamic adjustments would be made to dynamically adjust themotion scaling factor for the system. Also, as with the FIG. 6Aembodiment, motion scaling may include a filter on the monitoreddimension to ensure smooth transitions from different motion scalingfactors.

The embodiment illustrated in FIG. 8 is similar to those described withrespect to FIG. 7, and includes components allowing for identificationof instrument end effectors displayed in the intraoperative view.Referring to FIG. 8, input relating to the end effector positions in theintraoperative view is used by the system to determine the distancebetween each instrument and the center C of the intraoperative view, asdisplayed on the monitor. The motion scaling factor is selected oradjusted based on the relative distance between the instrument tip andthe center of the intraoperative. As with the first and secondembodiments, the third embodiment would preferably use filtration tofacilitate smooth transitions between scaling factors.

The process flow chart of FIG. 9 illustrates aspects of the first,second and third embodiments.

User inputs to the system for causing movement of surgical instrumentson the robotic manipulators are captured by the user interface. Themotion scaling factor is used to relate the motion of the inputs to themotion of the outputs. Note that the motion scaling factor is broken outas a separate entity from the other aspects of the software for relatingthe user inputs to command outputs for the robotic manipulators, inorder to make the flow chart clearer. The surgical instrument endeffector is moved by the robotic manipulator.

The camera captures images of the end effector in the operative fieldand those images are displayed to the user on the display. The systemthen identifies the end effector location(s) using techniques describedabove, receives any other inputs required in defining the vector(s)(i.e. gaze location via eye tracker), and defines the relevant vector(s)as described above. The system then applies the applicable scalingfactor based on the defined vector(s).

It should be noted that the distance-based scaling determinations may befurther combined with concepts described above in connection with theinstrument-based scaling. In other words, the database may correlatescaling factors with other data in addition to distance. Such other datamay include any combination of: the type or types of instruments whosedistance is being determined, procedure type, task type, and user.

The described system and method offer a number of advantages overexisting technology, including adjustment of the motion scaling factorto meet the needs of the surgeon at a given moment in the procedure.Additionally, increasing scaling at points in the procedure eliminatesor reduces the frequency of clutching for surgeon comfort by requiringless motion at the user input device to produce the desired motion ofthe surgical instrument.

Scaling Based on Surgeon Monitoring

In other embodiments, the system includes a sensor 34 configured tosense changes in a transient user parameter indicative of a change inthe user's focus or emotional arousal. In other words, the sensor mightbe one that senses parameters representing physiological or behavioralcharacteristics. For this purpose, the system includes a sensor 34positioned to monitor the user parameter. When the user is performing acritical task requiring high focus and small precise instrumentmovements, he/she may want the motion scaling to be set very low.However, at times the user will need to move an instrument a greaterdistance to reposition adjacent tissue or deal with a new and moreurgent situation. At these moments, the user may prefer a higher motionscaling factor to enable larger instrument motions without repeatedclutching of the input device.

In the disclosed system, the control system is configured to dynamicallychange the motion scaling factor based on data from the sensors whichmonitor the surgeon. The following concepts could be used tocharacterize the state of the surgeon and may offer opportunities forenhancing the user experience via dynamic motion scaling. These could bealone or in various combinations to create a maximally effective dynamicscaling control system.

1. User Blink Frequency

Decreased blink frequency may indicate an increased level of user focus.This heightened focus may be due to a critical moment in the procedureor reduced surgeon confidence. In this embodiment, the sensor is animage sensor positioned to capture images of at least one of the user'seyes. As one example, an eye tracker may be used. An image processingalgorithm is used to identify when an eye is open or closed, allowing adetermination of the user's blink rate. The blink rate is monitoredduring the procedure, and, in response to a detected decrease in blinkrate frequency, the processor causes the motion scaling factor to bereduced to help the user move more carefully, deliberately, andprecisely. Where the system includes an eye tracker 21 to give inputthat results in camera control or movement, or for some other purpose,that eye tracker may be used as the sensor 34 to generate input fromwhich the processor can determine blink rate.

2. Change in Heart Rate or Blood Pressure

Changes in user vital signs such as heart rate or blood pressure cancorrelate to the level of user focus and/or stress. In this embodiment,the sensor 34 is a heart rate sensor (e.g. body worn electrodes oroptical sensors) and/or blood pressure sensor. Using this informationmay enable the adjustment of motion scaling factors as the user's heartrate and/or blood pressure changes. The processor may be programmed todetermine, based on additional considerations or input from othersensors, whether the change in heart rate or blood pressure data shouldresult in an increase or decrease in the scale factor. For example, ifthe processor determines that an increase in heart rate has occurred,but the system measures very little increase in input handle motion andan eye tracking sensor 21 determines that the user's gaze is stillfocused on the same portion of the image displayed on the image display,the scale factor should be low. If the heart rate increases, the usergaze location changes and there is an increase in handle motion, scalingmight be increased.

3. Change in Respiratory Rate

Breathing rate is another vital sign that can be monitored similarly toheart rate. A respiratory rate sensor may be a body-worn sensor thatdetects deflections of the chest, such as strain gauge wearable on theuser's chest or abdomen. When used in harmony with other measurements,such as those described here, breathing rate could help to increase thefidelity of the dynamic scaling control system.

4. Tone/Volume of Voice or Frequency of Verbal Communication

In this aspect, the sensor 34 may be an auditory sensor. Typically,during critical parts of procedures, the OR gets noticeably quieter.This change in detected volume could indicate a high focus moment,causing the processor to respond by applying a lower scale factor. Asanother example, the sensor output could be analyzed for changes in thetone of the surgeon's voice, since such changes could indicate changesin his/her level of frustration or stress. Combining this informationwith other data from the control system could be very useful forautomatic adjustment of the scale factor.

5. Sweating/Galvanic Skin Response

In this embodiment, the sensor measures the electrical resistance of thesurgeon's skin. Detected changes in the electrical resistance can beused by the processor to alter the scaling rate, as these changes may beindicators of the user's stress level. When used in conjunction withother measurements, skin response could increase the fidelity of thedynamic scaling control system.

The described surgeon monitoring sensors may additionally be used asinput to a machine learning algorithm used in a surgical system in orderto implement and optimize adjustments to motion scaling based on senseduser parameters such as those described herein. The sensors would beused in the robot assisted surgical system to monitor the parametersindicative of the surgeon's stress and focus level. Data would also becollected correlating those parameters with the motion of theinstruments (as sensed, for example, using sensors measuring movementsof the user input device and/or sensors measuring movements of thesurgical instruments), the procedure being performed (as recognized bythe system or input by a user), motion of the input handles, motionscaling settings selected, etc. An analysis of this procedure data withdata on the sensed user parameters are thus used to build and optimize ahighly effective control system for motion scaling based on ananalytical model with surgeon physiological behaviors as a part of theinput.

The disclosed concepts provide advantages over existing technology.Current robotic systems use menus to select the desired motion scaling.During critical moments, this can be frustrating to the user becausethey are unable to effectively complete the desired task withoutswitching the motion scaling level which takes too much time. Thedescribed system causes smooth and effective motion scaling adjustmentautomatically such that the user would ideally not even notice a change.

All prior applications and patents referred to herein, including forpurposes of priority, are incorporated herein by reference.

We claim:
 1. A robot-assisted surgical system comprising: a roboticmanipulator configured for robotic positioning of a surgical instrumentin a body cavity, at least one primary user input device moveable by auser to cause the robotic manipulator to move the surgical instrument inthe body cavity, at least one auxiliary user input device operably by auser; at least one processor and at least one memory, the at least onememory storing instructions executable by said at least one processorto: receive user input in response to movement of the input device by auser receive instrument identification input in response to positioningof a surgical instrument on or in proximity to the manipulator cause themanipulator to move the first surgical instrument in response to theuser input, receive instrument type input based on signals from thesensor in response to positioning of a surgical instrument on, or inproximity to the robotic manipulator, and assign a function to theauxiliary input device based on the instrument type.
 2. The surgicalsystem of claim 1, wherein the at least one memory stores instructionsexecutable by said at least one processor to: receive input from a userspecifying a surgical procedure type, and assign a function to theauxiliary input device based on the instrument type input and surgicalprocedure type input.
 3. The system of claim 2, wherein, theinstructions are further executable by the at least one processor to:predict a subsequent task to be performed based on the specifiedprocedure type; receive instrument type input based on signals from thesensor in response to positioning of a second surgical instrument on, orin proximity to the robotic manipulator, and assigning a function to theauxiliary input device based on the instrument type input, the predictedsubsequent task, and the specified procedure type.
 4. The system ofclaim 2, wherein: the instructions are further executable by the atleast one processor to receive input identifying a user; and theauxiliary input device function is assigned based on the instrument typeinput, the specified procedure type, and the user identity.
 5. Thesystem of claim 1, wherein the auxiliary device is a foot pedal.
 6. Thesystem of claim 1, wherein the auxiliary device is a knob or button. 7.The system of claim 1, wherein the auxiliary device is an input handle.8. The system of claim 1, wherein the auxiliary device is a switch.